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S -Check,  by  Example 


Robert  Snelick 

Abstract:  S-Check  is  a software  tool  for  identifying  performance  bottlenecks  in  parallel  and 
networked  programs.  The  system  uses  and  incorporates  sophisticated  statistical  techniques 
and  synthetic  perturbation  for  extracting  code  sensitivities.  The  methodology  demands  exten- 
sive setup  and  execution  procedures.  S-Check  automates  much  of  this  process.  Even  still, 
using  S-Check  at  first  glance  can  be  formidable  because  of  its  unfamiliar  scheme  for  perfor- 
mance analysis.  To  alleviate  the  initial  learning  curve  we  present  a simple  walk  through  exam- 
ple of  how  to  set  up  and  conduct  an  S-Check  performance  analysis. 


Introduction 

We  describe  how  to  use  the  sensitivity  analysis  tool  S-Check  [3]  by  work- 
ing through  a simple  example.  This  step-by-step  procedure  can  be  used  to 
familiarize  yourself  with  S -Check’s  basic  window  layout  and  flow  (Figure 
1).  You  can  follow  along  on-line  by  going  to  the  example  code  directory 
given  with  the  S-Check  distribution  (release  2.0  and  later)  [2].  The  sample 
code  (a  simple  quicksort  program)  is  under  the  example  directory  in  the  top 
level  of  the  distribution.  Start  S-Check  while  in  this  directory.  As  we  go 
through  the  example,  user  actions  will  be  outlined  in  highlighted  boxes.  It 
is  assumed  that  the  reader  is  familiar  with  basic  S-Check  concepts 
[3, 4, 5, 6]. 

Below  is  a listing  of  the  steps  and  procedures  required  by  S-Check  to  com- 
plete its  analysis  of  your  code.  Some  steps  are  trivial,  requiring  just  a sim- 
ple user  response  or  are  completely  automated  by  S-Check.  Others  involve 
moderate  input  and  interaction  from  the  user. 

• select  experiment  name  (pick  a name  to  identify  the  experiment) 
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• identify  run  environment  ( e.g .,  IBM  SP2  using  LoadLeveler) 

• indicate  your  test  code  (your  application) 

• indicate  compile  and  runtime  instructions  (e.g.,  include  parallel  routine 
library) 

• select  type  of  S-Check  analysis  (e.g.,  a basic  screening  of  computational 
segments) 

• pick  suspect  code  locations  to  test  (where  do  you  suspect  performance 
bottlenecks) 

• define  response  interval  (e.g.,  overall  run  time  of  the  program) 

• select  DEX  plan  (trade-off  between  cost  and  information) 

• select  number  of  experiment  replication  (increase  for  a noisy  system) 

• select  delay  value  (duration  of  delay) 

• build  experiment  (compile  code  with  instrumentation) 

• run  experiment  (run  all  program  variants,  record  data) 

• calculate  effects  (solve  set  of  linear  equations) 

• view  results  (fist  or  plot  results  from  previous  step) 

• tune  code  accordingly  and/or  change  experiment  parameters  and  return 
to  a prior  step 
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Figure  1 . S-Check  Window  Layout  and  Flow. 
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Step-by-Step  Example 

gfgpt  Before  starting  S-Check  make  sure  that  it  is  installed  correctly.  This 

S-Check  includes  having  S-Check’s  executables  in  your  path  and  making  sure  that 

S-Check  is  reading  its  resource  file.  See  “How  to  install  S-Check”  in  Using 
S-Check  [1].  To  invoke  the  tool  just  type  scheck  while  in  the  example  direc- 
tory or  in  the  directory  in  which  your  code  resides. 


% cd  <your _^af/*>/scheck2.0/example 
% scheck  & 


Create 

Experiment 


Configure 

Experiment 


The  first  window  you  see  is  the  Experiment  List  Window  (Figure  2).  This 
window  allows  you  to  create  new  experiments  and/or  recall  previously 
saved  experiments.  We  refer  to  an  experiment  as  the  entire  process  (static 
and  dynamic  aspects)  that  defines  a particular  S-Check  analysis.  To  create  a 
new  experiment,  give  it  a name  and  click  on  Open. 


> type  in  test.l  in  the  New  Experiment  Name  text  field 

> click  on  the  Open  button 


SCttec*. 


Since  this  is  a new  experi- 
ment, S-Check  launches  the 
Configuration  Window  (Fig- 
ure 3)  where  the  platform 
type , test  code , and  experi- 
ment type  can  be  initialized. 

The  Platform  Type  indicates 
to  S-Check  what  environ- 
ment your  source  code  will 
run  on.  The  Platform  Type 
option  menu  allows  you  to 

select  your  platform.  For  Figure  2.  Experiment  List  Window 

simplicity,  we  demonstrate  the  tool  on  a uniprocessor  Unix  workstation. 
Therefore,  we  use  the  default  Platform  Type  (Unix).  This  requires  no  action 
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by  the  user.  Next,  we  tell  S-Check  which  code  we  are  going  to  test.  S- 
Check  lists  all  .c  files  in  the  current  directory.  Here  we  need  to  select  all  the 
files  that  will  make  up  the  executable  program.  Drag  and  hold  your  pointer 
over  the  files  listed  under  the  Directory  Contents  selection  window  to  high- 
light the  files.  Click  on  the  Add  button  to  include  them  in  the  Work  Set.  The 
Work  Set  contains  the  list  of  files  that  will  be  edited  for  the  purpose  of 
picking  test  points.  Any  flags  (or  other  attributes)  that  you  need  to  set  to 
compile  or  run  the  code  are  identified  next.  In  our  simple  example  the  only 
parameter  that  needs  to  be  filled  in  is  the  command  line  Arguments  text 
field.  Here  we  enter  1000000.  This  instructs  the  example  program  to  sort 
1000000  numbers.  We  enter  command  line  arguments  in  the  same  manner 
as  if  we  were  running  the  code  from  the  shell. 

> leave  Platform  Type  set  at  Unix  and  Experiment  Type  set  to  Screening 

> highlight  all  files  in  Directory  Contents  by  dragging  the  pointer  on  them 

> add  the  selected  files  to  the  Work  Set  by  clicking  on  the  Add  button 

> enter  1000000  in  the  Arguments  text  field 

> click  on  the  OK  button  to  exit  the  Configuration  Window 


Next  we  select  the 
Experiment  Type. 
There  are  4 options: 
Screening,  Barrier, 
Communication,  and 
Scaling.  In  our  exam- 
ple, we  just  want  to 
evaluate  the  impact  of 
certain  suspect  compu- 
tational code  seg- 
ments. Therefore,  we 
keep  the  default  selec- 
tion (Screening).  This 
will  run  S-Check 
basic  sensitivity  analy- 
sis. Clicking  on  OK 


Figure  3.  Configuration  Window 
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Select 

Test 

Locations 


exits  this  window  and  leads  us  to  the  Experiment  Control  Window. 


> open  bubble.c  by  double-clicking  on  it  in  the  Experiment  Control  Window 

> move  your  pointer  to  line  21  in  bubble.c  and  click  on  “swap(&list[i], 

> move  your  pointer  to  line  10  and  click  on  the  code  segment 

> click  on  the  OK  button  to  exit  the  editor 


Before  we  can  access 
many  of  the  functions  of 
the  Experiment  Control 
Window  (Figure  5),  we 
must  first  select  the 
code  segments  that  we 
wish  to  investigate  and 
define  the  response 
interval.  So  we  will  do 
that  first.  One  way  to 
select  points  to  screen  is 
to  use  S-Check’s  Factor 
Editor  Window  (Figure 
4).  In  the  upper  left  hand 
comer  of  the  Experiment 
Control  Window  is  the 
list  of  Work  Set  files  we 
previously  selected  in 
the  Configuration  Win- 
dow. To  open  a Factor 
Editor,  double-click  on 
the  desired  file. 


We  select  test  locations  Figure  4.  Factor  Editor  Window 

(called  factors ) by  simply  clicking  on  the  desired  code  segment.  This 
action  highlights  a section  of  text  where  you  clicked.  The  delay  instrumen- 
tation will  be  inserted  between  the  left  most  character  and  the  right  most 
character  of  the  reverse  video.  Some  exceptions  to  this  rule  for  certain  com- 
pound statements  exist,  see  Chapter  4 in  the  user’s  guide  [1]. 
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> open  parte  by  double-clicking  on  it 

> move  to  line  23  and  click  on  the  “more  to  do”  variable,  the  “ {“  section  of 
the  line  is  highlighted  (the  body  of  the  while  loop  will  be  tested) 

> click  on  the  OK  button  to  exit  the  editor 


Once  a factor  has  been  selected,  the  factor  count  is  updated  for  the  current 
file  that  is  being  edited.  A global  factor  count  for  the  experiment  is  main- 
tained on  the  Experiment  Control  Window  (Figure  5).  A fist  of  the  cur- 
rently selected  factors  are  shown  in  a panel  under  the  source  code  viewer. 
In  our  example  Factor  Editor  screen  (for  bubble.c)  we  selected  two  test 
points.  For  the  entire  program  we  will  select  three  locations  (two  in  bubble.c 
and  one  in  part.c). 


A Factor  can  be  removed  by  clicking  on  the  location  again.  The  text  area 
will  no  longer  be  highlighted  in  reverse  video.  Factors  can  also  be  selected 
in  groups  with  the  use  of  S-Check’s  automatic  factor  selection  utility.  See 
the  Profile  button  under  the  Utility  Menu  on  the  Experiment  Control  Win- 
dow. We  don’t  use  this  feature  in  our  simple  case  study,  but  it  can  be  very 
useful  in  large  programs. 


> open  main.c  by  double-clicking  on  it 

> click  on  the  Select  Response  button 

> scroll  down  to  line  40  and  select  it  (a  B is  placed  in  the  annotation  column) 

> scroll  down  to  line  44  and  select  it  (a  E is  placed  in  the  annotation  column) 

> exit  the  Factor  Editor  window 


Set 

Response 

Interval 


Next  we  need  to  define  the  response  interval  which  is  the  section  of  the 
code  we  want  to  improve.  This  response  measure  is  usually  set  to  record 
the  run  time  for  the  whole  program.  To  define  the  response  interval,  we 
open  up  a Factor  Editor  for  the  file  where  we  want  to  set  it,  usually  the 
main  driver.  When  started,  a Factor  Editor  by  default  is  automatically  set  to 
factor  selection  mode  (as  we  have  seen  previously).  To  change  this  mode 
we  use  the  Select  Response  button  (lower  right  on  the  Factor  Editor  Win- 
dow). This  button  is  used  for  setting  the  Begin  (B)  and  End  (E)  of  the  tim- 
ing interval.  Upon  clicking  on  the  Select  Response  button,  the  cursor 
changes  to  a red  downward  pointing  arrow.  This  symbol  indicates  that  the 
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Build 

Experiment 


Start 

Experiment 


Begin  can  be  set  (click 
to  mark  it).  After  the 
Begin  location  is  set 
the  cursor  points 
upward  and  is  ready 
for  defining  the  End 
point.  Another  click 
sets  the  End  location 
and  completes  the 
response  interval. 


the  test 
response 


Now  that 
points  and 
interval  are  defined, 
we  can  set  up  the  rest 
of  the  experiment.  We 
do  this  using  the 
Experiment  Control 
Window.  Here  we  can 
open  Factor  Editors  (as 


response  tine[  6]  is: 

5.52 

100 

response  tine[  7]  is: 

5.21 

000 

response  tine[  8]  is: 

7.76 

001 

Variance  conputed  uith  1 

degree (s)  of 

Freedon 

You  need  at  least  10  for 

a neaningful 

estinate 

: Variance  = 0.000215 

Standard  error  0.014676 

Fi  g u re5 

discussed),  select  a designed  of  experiment  plan,  set  the  number  of  replica- 
tions, and  determine  the  delay  value.  In  S-Check,  all  of  these  have  default 
settings  or  can  be  generated.  In  our  example,  we  use  the  default  settings. 


> click  on  the  Build  button  of  the  Experiment  Control  Window 


In  addition,  we  use  the  Experiment  Control  Window  to  build  and  start 
experiments.  To  build  an  experiment  we  click  on  the  Build  button.  This 
compiles  the  program  with  our  instrumentation  instructions.  Once  the  pro- 
gram is  built,  S-Check  runs  sample  tests  to  determine  a suitable  delay 
value.  The  progress  of  this  phase  (as  well  as  other  phases)  is  chronicled  in 
the  Messages  area  (bottom  of  window). 

Upon  completion  of  this  phase,  the  Start  button  becomes  functional.  We 
begin  the  experiment  by  selecting  this  button.  S-Check  proceeds  to  execute 
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each  program  variant  and  displays  various  status  information  on  the  Exper- 
iment Control  Window.  A running  trace  of  each  trial  run  is  shown  in  the 
Messages  area.  The  response  time  and  the  corresponding  delay  pattern  is 
displayed. 


> click  on  the  Start  button  to  begin  the  experiment 

> upon  completion,  select  the  List  Effect  button  from  the  Results  menu 


View 

Results 


Upon  successful  completion  of  the  experiment,  the  List  button  under  the 
Results  menu  is  enabled.  Select  this  choice  to  bring  up  the  List  Effects 
Window  (Figure  6)  to  view  the  sensitivity  analysis.  S-Check  displays  the 
effects  for  each  factor  defined  in  the  factor  selection  phase  of  experiment 
initialization.  It  can  also  display  the  interaction  effects  for  these  factors  (if 
available).  An  interaction  effect  indicates  the  affect  of  code  efficiency 
changes  on  two  or  more  factors  together. 


SCtecfi:  list  Wioctow 


&!perii»eafcj|teSt.i 


rHain 

j2tv} 

J3rd  Orster 
jfill 


Experinent  Profile: 
Experinent  : 
Plaffom  : 

Test  Type  : 
Factors  : 

Plan  : 

Replication: 

Std  Error  : 0.01 
Delay  Value:  1 


test.l 

Uorkstation 

Screening 

3 

Full  Factorial 


An  effect  reflects  the  impact  the  code  efficiency  changes  (the  delay)  had  on 
the  run  time  of  the 
program.  The 
higher  the  effect, 
the  more  likely  the 
corresponding 
code  segment  is  a 
bottleneck.  The 
tuning  effort 

begins  at  the  high- 
est ranked  code 
segment.  The  sig- 
nificance of  an 
effect  is  deter- 
mined by  its  rela- 
tive magnitude  to 
other  effects  and  a 
standard  error 


V Index 
Value 


EffecE  Order  Tern  [line  8]File 


[ 23]part.c  partition_list() 

[ 21Jbubble.c  tuible.sortO 

[ lOlbubble.c  biiible  sortO 


Figure  6.  List  Effects  Window 
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measure.  Effects  should  be  at  least  3 standard  errors  from  zero  to  be  con- 
sidered significant.  As  shown  in  the  List  Effects  Window,  each  effect  mea- 
sure is  related  back  to  a corresponding  section  of  code.  Double-clicking  the 
right  mouse  button  on  a code  segment  opens  a Factor  Editor  for  viewing 
the  code  segment  in  question.  Investigation  (improvement)  of  code  sections 
proceeds  down  the  ordered  list  until  a desired  performance  level  is 
obtained.  Alternatively,  further  exploration  of  the  program  can  be  per- 
formed by  discarding  some  or  all  factors  and  adding  more  factors  and  then 
retesting.  The  process  can  be  iterative. 

Interpretation 
of  Results 


The  3 factors  we  selected  to  analyze  are  for  demonstration  purposes  and 
correspond  to  the  operations  of  (1)  exchanging  elements  (using  swap())  in  a 
bubble  sort  routine  (2)  calls  to  the  bubble  sort  routine,  included  here  for  a 
baseline  measure  and  (3)  comparing  elements  in  a fist.  A typical  result  pro- 
duced by  S -Check  will  look  like: 


ID 

Effect 

Order 

Term 

[line  #]  File 

Function 

Text 

2 

2.43 

1 

2 

[23]part.c 

partition_list() 

while(more_to_do)  { 

0 

0.25 

1 

0 

[21]bubble.c 

bubble_sort() 

swap(&list[i],  &list[i+1]); 

1 

0.01 

1 

1 

[10]bubble.c 

bubble_sort() 

{ 

Standard  Error  = +/-  0.02 


Of  the  three  elements  that  we  decided  to  investigate,  the  main  comparison 
loop  (ID=2)  of  partition _list()  is  ranked  the  highest  in  terms  of  its  overall  per- 
formance cost.  This  is  to  be  expected  given  that  one  of  the  central  opera- 
tions involved  in  the  quicksort  algorithm  is  comparing  and  subsequently 
exchanging  elements  to  split  fists.  In  comparison  bubble_sort()  is  less 
costly  because  it  is  only  used  to  sort  subfists  of  sufficiently  (economically 
wise)  small  length.  In  bubble_sort(),  we  examined  two  code  areas:  swapping 
elements  (ID=0)  and  the  function’s  entry  point  (ID=1).  As  expected  (for 
our  data  set),  the  swapping  phase’s  impact  on  performance  exceeds  that  of 
the  function’s  entry  point  impact.  In  fact,  since  calling  the  bubble  sort  plays 
a minor  role  in  the  algorithm,  its  (ID=1)  influence  on  performance  is  barely 
noticed  in  S-Check’s  analysis.  In  contrast  the  key  component  of  the  sort, 
the  partitioning  phase  (ID=2),  is  resoundingly  brought  to  attention. 
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The  simple  example  presented  here  describes  S-Check’s  core  usage  and 
shows  how  to  apply  its  performance  analysis  to  tune  programs.  Although 
the  example  is  trivial,  the  same  procedure  and  analysis  just  as  easily  applies 
to  complex  programs  on  a variety  of  parallel  and  networked  architectures. 
The  tool  is  quite  robust— the  largest  code  it  has  analyzed  had  44,  000  lines. 


Save 

Experiment 
& Results 


Our  experiment  can  be  revisited  at  a later  or  more  convenient  time  by  sav- 
ing the  experiment  and  results.  Results  are  saved  under  the  Save  button  in 
the  Results  menu.  Experiment  settings  are  save  under  the  Save  button  in  the 
File  Menu.  The  use  of  these  mechanisms  can  aid  and  log  an  ongoing  per- 
formance evaluation  of  an  application. 


> select  Save  under  the  Results  menu  on  the  Experiment  Control  window 

> name  the  file  and  click  on  SAVE 

> select  Save  under  the  File  menu  on  the  Experiment  Control  window 

> select  Quit  under  the  File  menu  on  the  Experiment  Control  window 


Exit  S-Check 


To  exit  S-Check,  select  the  Quit  button  under  the  File  menu  on  the  Experi- 
ment Control  Window. 


Conclusion 

We  illustrated  S-Check’s  multistage  process  for  assessing  and  identifying 
performance  bottlenecks  in  parallel  and  distributed  codes.  The  intention 
was  to  introduce  the  user  to  the  basic  S-Check  concept  and  process.  For 
clarity  and  brevity,  many  features  and  options  were  intentionally  omitted. 
The  users  guide  and  related  publications  can  further  explain  S-Check 
usage. 
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Disclaimer 

The  National  Institute  of  Standards  and  Technology  (NIST)  contribution  is 
not  subject  to  copyright  in  the  United  States.  Certain  commercial  products 
are  identified  in  this  paper  to  adequately  specify  experimental  procedures. 
Such  identification  does  not  imply  recommendation  or  endorsement  by 
NIST,  nor  does  it  imply  that  the  products  identified  are  necessarily  the  best 
available  for  the  purpose. 
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